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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
and training designed to create new evaluation methodologies for 
use in education. This document is one of a series of papers and 
reports produced by program staff* visiting scholars* adjunct 
scholars, and project collaborators—all members of a cooperative 
network of colleagues working on the development of new 
methodologies • 

How can evaluators determine the feasibility of using a new 
evaluation method for a given problem? What criteria can be used 
in judging the utility and effectiveness of a new method in 
making both formative and suiranatiye assessments of the method's 
quality? Answers to these questions are provided in this report 
through a discussion of five checklists designed to aid 
evaluators in conducting field trials of new methods. 

Nick L. Smith, Editor 
Paper and Report Series 



FOREWORD 



The substance of this report draws upon and summarizes work 
completed to date by the Research on Evaluation Program (ROEP) in 
the field, testing of alternative evaluation methods,. This 
report^ along With an earlier companion piece (Guidebook to New 
Evaluation Methods, Caulley and Smith* 1981, ROEP Report Number 
66) , is designed to aid evaluation practitioners in the selec- 
tion, adaptation, and implementation of alternative evaluation 
approaches. These two reports will be used during the 1983 
contract year in providing consultation services to evaluation 
practitioners. A combined, revised report will be prepared in 
1984 based on this field use. The Research on Evaluation Program 
staff welcome reactions, comments and revision suggestions 
concerning both reports. 
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A GUIDE FOR J30NTOCT1NG FIELD TRIALS 
OF NEW METHODS 



Trial Criteria and Check Lists 



In recent years there has been increasing interest in the 
development of new methods for conducting program evaluations. 
This interest has been stimulated by problems of matching 
existing methods to diverse evaluation contexts and problems to 
provide results that are useful to decision makers. As a result, 
e valuators have explored geography, epidemiology, law, 
philosophy, history, art, anthropology, economics, journalism, 
and many other areas in search of new perspectives and tools for 
evaluation. (cf. Smith, 1981 (a), 1981 (b) , 1981 (e)) 

As new approaches have been identified and described, an 
expanded repertoire of methods has become available for 
adaptation in evaluation settings. However, since few methods 
have yet been used for program evaluation, questions remain about 
their practical worth. These questions focus on the feasibility , 
utility , and effectiveness of the new methods. 

One of the best ways to determine the worth of a method is 
through its implementation in a field trial. By a field trial, 
we mean any adaptation of. a method in a field setting. Field 
trials may take various forms: 

• In situ Study — 'Study of a naturally occurring use of a 
specific evaluation method; 

• Pilot Study — field trial of a new method or of an 
existing method in a new context* a trial designed to 
study the method, not to do an evaluation; 

• Application S tudy — conduct of ah evaluation employing a 
new method not used before in a similar context, where 
the evaluation places only secondary attention on the 
study of the method? 

• Comparative Study — an evaluation employing two or more 
methods, enabling comparisons to be made among them; 



The overriding question which should* guide a field trial is, 
"How well does the method work?" In terras of oar three major 
criteria, the question is modified to be, " How well is the 
method likely to work?" (feasibility), "How well is it working?" 
(utility and effectiveness in a formative sense), and "How well 
did it work?" (utility and effectiveness in a sui.imative sense). 
The purpose of field trials is to bring empirical, practical * and 
social value concerns to the fore in the development of new 
methods. This guide, then, provides practitioners interested in 
new methods with a framework for determining the feasibility, 
utility, and effectiveness of the methods in field settings. 

To accomplish this intent, the body of this guide contains 
discussions of five checklists. Three of the checklists are to 
be used before a trial to determine the feasibility of one or 
more methods, and during and after a trial to determine the 
method's utility and effectiveness. These checklists all focus 
on the same elements, that is, various aspects of an evaluation 
such as its purpose, audience concerns, setting constraints, 
major questions, data collection and analysis methods, and 
reporting. The checklists' questions are phrased so as to 
explicate the criteria of feasibility, utility, and effectiveness 
at the three stages of trial* namely, during planning, in 
process, and upon completion. 

An additional checklist, lists factors related to ideal field 
trial conditions. Deviations from these ideal conditions suggest 
the limitations that may be imposed on a method in a particular 
field trial. The limitations have to do with expectations for a 
successful field trial arid expectations regarding the information 
obtained about the method's worth. 

A final checklist contains an outline for recording the 
conditions and results of a field test. 

The purpose of this guide is to provide practitioners with 
practical method review guidelines, therefoe, more technical 
discussions of the development and testing of methods have riot 
been included here. (See, for example, Smith 1979, Smith 1981 
(a), Smith 1981 (e) , arid Smith 1982 (a).) A brief summary of 



the 27 methods trials performed to date by the Research on 
Evaluation Program (ROEP) has been included as Appendix A for the 
reader's future reference. A wide variety of method trials are 
presented in Appendix A, providing the reader with a range of 
alternative test strategies to consider. 

Appendix B of the guide contains on updated catalog of new 
evaluation methods. Thirty-eight new methods are described in 
terms of their purpose , basic procedures, advantages/benefits, 
disadvantages/costs, and resources required. The basic 
references provided for each method enable the reader to learn 
more about particular methods of interest. 

Feasibility and Ideal Test Conditions 

The more new methods there are, the more difficult it becomes 
to decide which one (s) to adapt for a field trial. Questions 
arise concerning the potential worth or feasibility of a method 
within the constraints of ah evaluation's context. 

The feasibility checklist (Checklist 1) contained on the next 
page can be used with the catalog of new methods (Appendix B) . 
For example, to determine if a method is consistent with the 
purpose (?) of the evaluation (Item I) , one would review the 
method v s purpose . The information related to why and when to use 
a method is useful in answering Item 2, will the method probably 
provide results that are appreciated by major interest groups? 

Reviewing the basic procedures of a method will help a 
potential user decide if the evaluator (s) can learn the method 
sufficiently to conduct a credible evaluation (f*-em 3). 
Reviewing this part of the method description will also reveal 
the extent to which the method has been developed. The basic 
references section provides additional information in this regard. 
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CHECKLIST 1: FEASIBILITY 



1. Is the method consistent with the purpose (s) of the 
evaluation? 

2. Does the method promise to answer the evaluation 
questions posed by the major interest groups? 

3. Can the evaluator (s) learn the method sufficiently well 
to conduct a credible evaluation? 

4. Is it possible to adapt the method within the 
constraints of the evaluation setting and still use it 
validly? 

5. Is there a need for this method, that is r does it 
P rov *^ e a _ a ??? ue approach/ or are there 

better methods? 
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It is very difficult to decide before the fact if it is 
possible to adapt a method within the constraints of the 
evaluation setting and still use it validly (item 4). However* 
the list of factors that provide for ideal test conditions 
(included here as Checklist 2) provide a framework for analyzing 
the setting in order to make that decision. The first five 
factors concern the context of the evaluation, that is, agency 
support, program clarity, conflicting influences and so on. 

The evaluator* s capabilities are assessed relative to the 
method in factor 6, and the extent of the method's development is 
considered in factor 7. 

Factors 8 and 9 focus on the interaction of the method and 
the program; the match between them in a general sense (8) and 
the availability of sufficient resources to implement the method 
(9). A careful reading of the why and when to 4ise , advantages/ 
benefits and disadvantages/costs sections of the method 
description can give the e valuator an idea of the likely match 
between the evaluation context and the method . From these 
sections and the resources— required section, come an 
understanding of the resources needed to adequately implement the 
method. 

The information gathered relative to these factors will 
indicate the degree to which trial conditions vary from the 
ideal. It will be up to the evaluator to determine if the 
deviation is sufficient to decide against trying the method being 
considered. The final item on the feasibility checklist, 
regarding the uniqueness of the method, requires that the 
evaluator look not only at new methods but at existing methods as 
well. In the final analysis, it may simply be unnecessary and 
unwise to try a new method when a familiar method will do. 



CHECKLIST 2: 
FACTORS THAT PROVIDE FOR IDEAL TRIAL CONDITIONS 



1. The agency sponsoring the evaluation should be 
supportive of trying the new approach. 

2. The project or program being evaluated should be clearly 
defined and fully implemented; 

3. There should be no conflicting political,, social, or 
organizational influences shaping the evaluation. 

4. The setting should allow maximum flexibility for 
adapting the method to local conditions. 

5. There should be internal support by the program staff or 
product developers for the evaluation of the new method; 

6. The test should be conducted by seasoned practitioners 
familiar with the hew method and with the evaluation of 
similar programs or products in such settings. 

7. The method should be as completely developed as possible 
prior to the empirical test. 

8. There should be an optimum match between the method , the 
program or product being evaluated, and the evaluation 
context. 

9. There should be ample resources (e.g. , time, money, 
people , materials) to conduct the evaluation using the 
new method • 
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Formative Utility and Effectiveness 



Once a method has been chosen, its use in a field trial must 
be monitored to determine how it is performing and to determine 
what modifications need to be made to make it effective. The 
items on the formative utility and effectiveness checklist 
(Checklist 3) are based on the standards associated with utility 
developed by the Joint Committee (1981). They concern in order: 
evaluator credibility, information scope and selection, audience 
identification, resource utilization, and evaluation impact. The 
last item provides a check on the fidelity with which the method 
is being used. 

Checklist 3 may be used by the evaluation staff or by others 
to conduct a formative meta-evaluation of the method. It can be 
helpful to have someone not directly involved in the use of the 
method act as a critic since, in their enthusiasm for using the 
method, the evaluators may lose sight of the larger picture 
represented by the utility and effectiveness checklist items. 

Sunroative Utility and Effectiveness 

At the conclusion of a field trial it is appropriate to take 
a retrospective look at the method and its implementation. The 
summative utility and effectiveness checklist (Checklist 4) has 
items on it which, when answered, generate the information to be 
included on the Field Trial Report form (Checklist 5) . There are 
six categories of information on each of these two instruments. 

ideal conditions factors . Before the field trial, factors 
that provide ideal conditions were considered in order to help 
determine the feasibility of trying a particular method. 
Reviewing the factors again at the end of a trial is a good 
starting point for assessing the method's worth since the 
deviations from the ideal indicate the context in which the trial 
took place. 
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CHECKLIST 3: FORMATIVE UTILITY AND EFFECTIVENESS 



1. Is the method being used by the evaluator (s) in a 
competent way? 

2. Does the method facilitate the collection of information 
sufficient to address the pertinent questions of the 
evaluation? 

3. Does the implementation of the method provide initial 
findings that are understandable by the major interest 
groups? 

4. Are the resources being used by the method consistent 
with initial projections: 

- time 

- cbst_ 
= People 

- materials 

5. What are the unanticipated outcomes or side effects of 
implementing the method? 

6. Is the implementation of the method remaining true to 
its basic assumptions or principles? 
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CHECKLIST 4: SUMMATIVE UTILITY AND EFFECTIVENESS 



What deviations from ideal conditions were present in 
the trial? 

a. bid the method serve the desired purpose? 

b. Were the important evaluation questions answered by 
the method? 

a. Did the data collection and analysis produce 
technically accurate results? 

b. Were the collected results analyzed and summarized 
in a way appropriate to the problem? 

c. Were the results of the evaluation useful to the 
major interest groups? 

Did the method produce an evaluation that met the 
resource conditions of 

- time 

- cbst_ 

- people 

- materials 

a. Was the method relatively easy to learn by the 
evaluator (s) ? 

b. Was the method sufficiently adaptable to be rated 
effective in this evaluation setting? 

c. What were the method's major strengths and 
weaknesses in this context? 

d. What were the unanticipated outcomes or side 
effects of using the method? 

e. Are there recommendations that should be passed on 
to other potential users? 



CHECKLIST 5: NEW METHOD FIELD TRIAL REPORT OUTLINE 



Method Name 

Program/Product (brief description) 

Trial Conditions Profile 

Program factors 

Evaluator capabilities 

Method development 

Method/Program match 

purpose 
procedures 

Resource Utilization 

time 

money 

people 

materials 

Results' Quality 

Technical accuracy of data collection and analysis 
Appropriateness to the problem 
Usefulness to major interest groups 

Method Analysis 

Ease of learning and use 

Adaptability 

Streng ths/weaknesses 

Unanticipated outcomes or side effects 
Recommendations for future users 

Field Trial Contact Person 

Other Contact People or Resources 
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Program related elements constitute the first five factors. They 
concern such things as agency and internal support^ program clarity, 
conflicting influences, arid flexibility. There may well be a change in 
one's analysis of these factors after a trial has been run. Resistance 
may have developed, conflicts may have arisen, and old prejudices for 
particular methods may have reduced the amount of flexibility available 
for adapting the methods. 

By referring back to the factors, an evaluator can generate a 
test's profile as shown on the Field Trial Report (Checklist 5). ' Under 
the method/program match, the evaluator is encouraged to give a brief 
summary of the purpose and procedures of adapting the method within the 
given setting. The use of resources is summarized under the last factor. 

Results 1 quality i The quality of information resulting from the 
use of a method is an indication of the method's utility and 
effectiveness within a given setting. By looking at the results, one may 
infer the worth of the method which produced them. There are three 
things to consider in assessing the quality of results; The following 
concepts of quality are based on the accuracy standards of the Joint 
Committee (±981) . 

The first aspect of quality has to do with the technical accuracy 
of the datai Of concern here is the validity and reliability of data 
collection, the system for managing data quality control, and the 
appropriateness of the analysis procedures to the evaluation problem. 

The second aspect of quality is the appropriateness to the problem 
of the information collected, analyzed, and summarized^ The basis for 
determining appropriateness lies in looking at the data within the 
context of the evaluation. The description of the trial conditions 
provides this perspective. 

Finally, the usefulness of the results to major interest groups 
must be considered in determining their quality. In this regard f 
conclusions must be justified in the eyes of decision makers and the 
reporting of results must be viewed as being objective. 
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Method analysis . The final set of checklist items arid the final 
set of report topics focus on the more complete description of the 
method, included here are the ease of learning and using the method, its 
adaptability, strengths and weaknesses, and any unanticipated outcomes 
and side effects resulting from the use of the method. 

Information from this last part of the checklist and report should 
help a prospective evaluator make a final decision about trying a 
method. For example, the ease with which a method is learned and used, 
together with the information about the evaluator f s capabilities 
presented earlier in the report, should help people judge if the method 
is right for them. In addition * an assessment of the extent to which a 
method can be adjusted in the process of making it effective will provide 
insight as to its robustness. The potential user can refer to this 
information after analyzing his or her capabilities, the evaluation 
setting, and the "strain" that these will put on the method as part of a 
feasibility check. 

After noting strengths and weaknesses of the method, unanticipated 
outcomes and side effects and any recommendations for future users, an 
evaluator can record more personal insights and anecdotes. These give 
the flavor of a trial and are often the most interesting parts of a 
report. 

The name and address of the field trial contact arid any other 
resources that the evaluator has become aware of as part of the field 
trial are included at the end of the Field Trial Report. 

Until there is concrete information about the feasibility, utility, 
and effectiveness of new methods, evaluation practitioners must make 
choices based oh superficial characteristics and projected benefits. 
What is needed is a body of literature describing in a consistent way 
information, from field trial experiences, about the worth of methods. 
It is hoped that the checklists provided here will help practitioners 
select, implement and evaluate the quality of new methods as well as 
provide documentation about their experiences that can be shared with 
other practitioners. 
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Stenzel Committee Hearings as an Evaluation Format 
{Available in Smith > 1982 (a) 5 

What is the nature of committee hearings as used in 
congressional inquiry and policy formation? Can 
committee hearings be adapted for use in educational 
evaluation? This report discusses these and other 
questions in assessing the potential of committee 
hearings as a tool in evaluation. Included in this 
report is a sample application of a committee hearing 
procedure in evaluation and ah extensive discussion 
of the various elements of the committee hearing 
process, including committee operation, staffing, and 
procedural rules. 

A. Templin Photography as an Evaluation Technique 
(Available in Smith, 1982 (b) ) 
(Prior version available as Report #32) 

Probably the most difficult aspect of developing new 
approaches devaluation is in conducting the first 
meaningful applications of them in field settings. 
This document contains a report of a field trial of 
the use of photography as an evaluation method ; A 
photographic study of a professional conference was 
conducted. Although the study was accomplished under 
severe time restrictions, this report illustrates the 
considerable promise of photographic evaluations i 
This document, written to help others who wish to 
conduct photographic studies, includes discussions of 
the working theories that can be used to_??id^_t± e 
photo taking - f the technical details and practical 
problems of concern in such a study, as well as the 
results of this particular conference photo study. 

£. Smith Meta-Evaiuation^ kite r native Perspectives 
(Available in Smith 1981 (bj ) 



What range of approaches are available for the 
evaluation of evaluations (meta-evaluation) ? Will 
differing approaches to meta-evaluation result in 
differing assessments of an evaluation's worthiness? 
These and related questions are addressed in this 
report which presents four comparative evaluations of 
the same report: "Some Still do: River Acres, Texas" 
by Terry Denny. The concept of meta-evaluation is 
significantly broadened by these roe ta-eva.luat ions 
which use water col or painting as a perspective 
(William J. Gephart), film criticism as a perspective 
(Gabriel Della-Piana) , philosophic analysis as a 
perspective (b. Bob Gowin^, and product evaluation as 
a Perspective J Jason Millman) . These are stimulating 
alternatives to consider as the need for quality 
meta-evaluations becomes more prevalent. 
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L. Smith Investigative Tracking as a Method of Library 
Evaluation 

(Available in Smith, 1982 (a) J 

Journalists have developed a simple strategy for 
using archival records in conducting evaluative 
investigations. This strategy provides a 
straightforward , cost-effective means of evaluating 
library operations and is especially suited to 
informal studies of small special libraries. 

E. Nelson investigative Journalism Methods in Educational 
Evaluation. 

(Available in Smith, 1982 (a)) 



How might one apply the methods of investigative 
journalism in evaluation studies performed in state 
education agencies? How would such methods compare 
with the more traditional approaches to evaluation? 
Dr. David Nelson of the Utah State Office of 
Education addresses these concerns by _ Presenting two 
comparative cases which illustrate the strengths and 
weaknesses of investigative journalism methods in 
state department settings. Dr. Nelson uses the newly 
developed stancferds-f or E duc ational Evaluation as a 
yardstick for assessing the relative merits of this 
new approach over conventional methods. 

McGaw The Use of Exploratory Data Analysis in 

Educational Analysis 
(Available in Smith, 1981 (b)) 



What is the nature of exploratory data analysis, and 
how does one perform such analysis? How might these 
techniques be used in educational evaluation? Barry 
McGaw considers these questions in this paper. tJsing 
^?§?_^? 01 ? fY?ly?^ion studies in education. Dr. McGaw 
leads the reader through a step-by-step summary of 
the basic techniques employed in exploratory data 
analysis. With the aid of this paper, evaluators can 
begin to shift their perspective from confirming 
assumed relationships to exploring possible 
relationships in educational programs. 
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L. Smith and D;N; Gaulley The Evaluation of Educational 
Journals Through the Study of Citatj/ons. 
(Available in Smith and Caiiliey, 1981) 

ft citation study of all major articles published in 
AERJ , RER , and ER in 1972 and 1973 was conducted to 
evaluate the impact of these journals on the 
educational literature. RER was found to be making 
the most significant impact in terms of subsequent 
citations of articles. While the analysis revealed a 
relatively closed network of inter journal 
communication within education for these articles, no 
discernible yearly citation pattern was identified 
for the individual articles. 



L. Smith Classic 1960s Articles in Educational Evaluation 
(a citation analysis application) 
(Available in Smith, 1981 (c) ) 

Which of the early 1960s literature in educational 
evaluation is still being widely read today? A 
citation study was conducted to identify which 
articles on evaluation theory and methods written in 
the 1960s continued to be heavily read and cited 
through the 1970s. While most of the writing of the 
1960s has ceased to have wide appeal, five classic 
articles were identified which continue. to 
significantly influence thinking in evaluation. 

A. Lane Using the Tools of Philosophy: Metaphor in 
Action 

(Available in Smith, 1982 (a)) 



How might the methods of philosophy be applied to the 
eva luation of educational programs? In what ways 
would evaluations using philosophical concept 
analysis look different from traditional studies? 
This paper presents a retrospective analysis of the 
possible use of philosophic techniques in a major 
reading evaluation conducted in a middle-sized school 
district. By working with the district evaluation 
personnel, the author of this report explores how 
that evaluation might have been different if the 
evaluation staff had employed philosophic procedures. 

Stanf ield Management Review of Evaluation Practice 
(Available as ROEP Report #58) 

What is the nature of evaluation practice in state 
departments of education when viewed by a management 
consultant rather than a traditional educational 
researcher? What management perspectives might be 
used to better understand the operation of state 
department evaluation units? These and related 
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questions are answered in this report by Dr. Jonathan 
Stanfield, an independent management consultant who 
conducted a review of the state evaluation units in 
Washington; California, and Montana. This report 
provides an illustration of a management consulting 
review study in evaluation as well as offering 
insightful observations about the nature of 
evaluation practice in state departments. 

11. J. Stanfield Pilot Field study of SEA Evaluation Costs 

(Available as R0S> Report #69) 

What are the costs associated with evaluation at the 
state level? Can management consulting techniques be 
used to be used to reduce some of these costs? After 
summarizing the costs of evaluation today 
(approximately $32 million at the state level) 
Dr. Stanfield reviews the ways in which management 
consulting techniques could replace standard 
evaluation approaches or save as much as $2.9 
million. Important distinctions in considering 
evaluation costs are also included in this pilot 
study report. 

12. J. Stanfield Management Consulting Case Study 

(Available as RDEP Report #74) 

Can management consulting techniques provide cost 
effective alternatives to standard evaluation 
approaches? What is the cost effectiveness of using 
management consulting techniques in SEA evaluation 
unit settings? This field study addresses these 
questions and concludes that management consulting is 

a competitive approach , but due to the limited 

applicability of management consulting to the study 
exercises selected and the lack of controls for 
comparison, it is not possible to show conclusively 
that management consulting is either superior or 
inferior to standard evaluation approaches. 

13. H. M. Levin and W. Seidman Manual for Cost Analysis in 

Educational Evaluation 
(Available in Levin, 1983) 

How does one select the appropriate cost analysis 
methods for use in an evaluation study? What are the 
procedures for estimating costs and benefits and for 

combining them to provide suitable cost analysis 

information? These and related issues are aiscussed 

in this report which is an extension of an earlier 

document (ROEP Report #33, Case Book on Cost Analysis 
in Educational Evaluation). This manual provides 
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training in performing OTst-benef it* cost effective- 
ness, cost utility* and cost feasibility analysis and 
includes major sections cn establishing a decision 
framework; identifying^ measuring^ and distributing 
costs; and assessing and aggregating benefits, 
effects, and utilities. 

14. G. M. Della-Piana Film Criticism and Micro-Computer Couseware 

Evaluation 

(available in Smith 1982 (a)) 

How might tectaiques from film criticism be used in 
the evaluation of instructional materials? Based on 
previous project work on methods of criticism, this 
report suggests an innovative system for providing 
??T^?P*?!?_ reviews of micro-*coraputer--based 
instructional packages. This system, which 
emphasizes the illumination of curricular data 
??^ a ^9 ris _^ ir ? u ?^ eri 9 a 95 ri 5 criticism techniques is 
designed to help audiences see how a piece of 
courseware works and to involve audiences in making 
their own interpretations and judgments of it. 



15. G. M. Della-Piana and C. K. Della-Piana Making Courseware 

Transparent: Beyond Initial Screening 
(Available as BDEP Report #76) 

How can one evaluate microcomputer courseware in a 
way that helps others make their own judgments of its 
quality? What alternative styles of evaluative 
presentations are available? This report answers 
these and related questions by describing a 
collection of procedures, with illustrative examples, 
for selecting and portraying evaluative information. 
A checklist for conducting microcomputer courseware 
critic ism is also included. 

16. B. E. Wholeben and J. M. Sullivan Multiple Alternatives 

Modeling in Determining Fiscal Roll-Backs During 

Educational Funding Crises 

(Available as ROEP Report #70) . 

How can one determine the proper mix of educational 
programs to receive reduced funding when budget 
cutbacks are necessary? This report provides an 
extensive discussion of the use of criterion 
referenced, mathematical modeling procedures to 
determine which budget reductions minimally reduce 
the quality of educational programs. Part I of this 
report explains the basic design of multiple 
alternatives analysis and the context for its use; 
Part II provides the technical and mathematical 
details of the analysis; and Part III contains an 
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extensive example of the use of _ these procedures in 
reducing a budget within a local school, district. 
This report describes a highly technical but workable 
solution to the difficult problem of reducing school 
budgets. 

17. A. Rasp, Jr. Interviewing to Augment Large Scale Survey 

Data: The Washington High School and Beyond 
Story 

(Available on ROEP Report #71) 

Is interviewing using trained volunteers a 
cost-effective way of providing supplemental 
data to increase the local relevance of national 
survey information? Can small-scale studies bo 
effectively piggy-backed onto larger national 
efforts to increase their local utility? Based 
9?_§ke project described here, the answer to 
both these questions is yes. This project, 
mounted by the testing and evaluation unit of 
the Washington state department of education, 
illustrates a cost-effective way to gather 
supplemental qualitative and quantitative 
information of increased local utility by 
properly augmenting a national 
study~ illustrating an important design option 
in this period of decreasing evaluation 
resources- 



Applications of Alternative Evaluation 
Communication Techniques 
(Available in Smith 1982(b)) 
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D. 


J. Macy Research Briefs. 


19. 


W. 


E. Hathaway Graphic Display Procedures. 


20. 


G. 


Leinhardt and S. Eeinhardt S tem-and-Leaf Displays. 


21. 


B. 


E. Whoieben Operational Network Displays. 


22. 


N. 


it. Smith Geographic Displays. 


23. 


M. 


Hendricks Oral Policy Briefings. 


24. 


R. 


E. Stake and D. E. Balk Briefing Panel Presentations. 


25. 


M. 


Levine Adversary Hearings. 


26. 


N. 


Stenzel Committee Hearings. 


27. 


J. 


S. Shoemaker Television Presentations. 
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JReglsed Cata l og of Metho d -Descriptions 



Page 



Data Collection and Analysis Strategies 

1. Assignment Models. ••••••••••••••• 27 

2. Transportation Models. ••••••••••••• 28 

3. Dynamic Programming. • ••••••••••••• 29 

4. Queueirig Theory. . • •••••••••••••• 30 

5. Minimum-Maximum Goal Projection 31 

6. Geocode Analysis •••• ...... 3 2 

7. Trend surface Analysis ............. 33 

8. Social Area Analysis/Ecological Analysis • • • • 34 

9. Concept Analysis • •••••••• 35 

10. Thematic Matrix Analysis 36 

11. Document Analysis 37 

12. Legislative History 38 

13. The Key Interview. . . . ........ . . ...... . . 39 

14. Interviewing: Circling, Shuffling and Filling . 40 

15. Documenting Files and Summaries. •••••••• 41 

16. Cost-Benefit Analysis. . 42 

17. Cost-Effectiveness Analysis 43 

18. Cost-Utility Analysis. ............. 44 

19. Cost-Feasibility Analysis. . 45 

Sampling Strategies 

20. Blanket Sampling • 4 6 

21. Shadow Sampling. 47 

22. Time-Based Sampling. ••••••••• 48 

23. Event-Based Sampling . ............. 49 

24. Dimensionally Based Sampling ••••• 50 



Reporting Strategies 

25. Research Briefs 51 

26. Appeals Procedures 52 

27. Storytelling 53 

28. Compelling the Eye . ._ 54 

29. Representation of Reality 55 

30. Accurate, Sharp Descriptions 56 * 

31. Graphic Displays 57 

32. Stem and Leaf Displays and Box Plots •••••• 58 

33. Still Photography 5? 

34. Oral Briefings 60 

35. Briefing Panel Presentations 61 

3 6. Adversary Hearings 62 

37. Committee Hearings 63 

38. Television Presentations of Hearings 64 
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Method s Assignment Models from Operations Research 

Purpose ; To assign something to something so some rairairaura or maximum is 
achieved. 

Why and when to Use ; Examples of when to use the assignment model are as 
follows: Tutors are to be assigned to students so that tutors' preferences 
are either minimized or maximized. Teachers are to be assigned to courses so 
that the teachers' preferences are either maximized or minimized. We have 
buses at various locations and we want to minimize the miles traveled to pick 
up pupils at various sites. 

Basic Procedures ; A simple algorithm is available to solve the problem by 
hand* 

Advantage s /Bene fits ; An advantage of the assignment model is that it gives a 

solution than can be obtained by inspection. The algorithm given for 
solution of the problems can readily be solved by hand without the problem of 
gaining access to a computer. A solution is available in a very short time. 

Disadvantageg/Costs s The algorithm is a little tricky to learn* 



Resources Required ; None. 



Basic-References : 

Caulley, D. N. The use of assignment and transportation models in 

evaluation * No. 68 of the Paper and Report Series of the Research on 
Evaluation Program* Portland, OR: Ncrthwest Regional Educational 
Laboratory, 1981. 

Eck, R. D. Operations research for business . Chapter 8. Belmont, CA: 
Wadsvorth, 1976. 

Hillier, F. S., & Lieberman, G* J. Introduction to operations research. 
Chapter 6. San Francis cot Holdeh-Day, 1967. 

Trueman, R* E. An introduction to quantitative methods in decision 
making . Chapter 8. New york: Holt, Rinehart and Winston, 1977. 
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Method : Transportation Models 

Purpose : To assign teachers to courses when courses each have a number of 
sections and teachers each nave a number of class periods available.. Note 
that this is different from the assignment model where each course has one 
section and each teacher has one class period available. 

Why and - When- to-4Jse : Suppose there are a number of courses to be taught and a 
number of teachers to do the teaching. These two numbers need not be equal. 
The teachers have available various numbers of class periods. There are a 
number of _ sections for _each_ course that are required to be taught. In order 
to ajjply the raodelr the total number of class periods, available must equal the 
total number of sections that are required to be taught. For each course the 
teachers have given a.preference rating according _to_whether_ taey_ would like 
to teach a course. For example , a seven-point rating might be used* with 1 
representing high preference and 7 representing low preference. The problem 
is to assign teachers to sections and courses so as to minimize the sum of the 
ratings* 

Basic Procedures : An algorithm is available to solve the problem by hand. 

advantages /Bene fits: An advantage of the transportation model is that it 
gives a better solution than can be obtained by inspection. The algorithm 
given for solution of the problems can readily. be solved by hand without the 
problem of gaining access to a computer. A solution is available in a very 
short time. 

Disadvantages/Costs ; The algorithm is a little tricky to learn. 
R e source s Requir ed : None • 



Caulley, D. N; The use of assignment and transportation models in 

^vaJ^a^ion . No. 68 of the Paper and Report Series of the Research on 
Evaluation Program* Portland , OR: Northwest Regional Educational 
Laboratory, 1981. 

Eck, R. D. Operations research for business . Chapter 8. Belmont/ CA: 
Wadsworth, 1976. 

Hillier, F. S., & Lieberman, G. J. Introduction to operations research . 
Chapter 6. San Francisco: Holden-Day, 1967. 

Trueman , R. E. An introduction to quantitative methods in decision 
making . Chapter 8. New York: Holt, Rinehart and Winston, 1977. 




3. 
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Method: Dynamic Programming 

Purpose ; To help a decision maker make a multi-stage decision, one that 

requires several interrelated choices* 

Why-and-When— to ^gse : Dynamic programming is useful in curriculum design. 
Suppose we have a mass of materials or topics, all ire possibilities for _ 
inclusion in a course of study, but we are limited in time. How do we cull 
out what is unnecessary? How dp we give the_ remaining material its proper 
weight, so that all will fit within the total time allotted, and achieve the 
raaximum_possible benefit?_ _ This_ is a familiar problem, and a subjective 
examples are: a university unable to decide what should be contained in a 
"liberal education;" a teacher of 0. S. history somehow unable to teach past 
the Civil War; and a district curriculum committee debating over which 
readings should be included in tenth grade literature; 

Baste Procedures : There are at least five requirements which appear necessary 
to solve the "curricular problem." 

1. A list of curriculum Materials. This list might be of classes which could 
be desirable for a training sequence; reading assignments for a history 
course; homework problems for general mathematics; field trips or films 
for science. 

2. Values for each topic. These would be subjective ratings, such as might 
be assigned by an experienced teacher. 

3. Costs for e ac h topic. These may be expressed in terras of any_ resource 
which is limited. In curricular cases, the most usual constraint is 
student time. 

4. Maximum cost permitted_for all topics . Usually there is only a limited 
amount of time available for the total intended course of study. 

5. A solution syst e m . To select a final set of topics from the candidate 
list (1 above), we need some way to combine the values of each topic (2) 
and its cost (3), such that the sum of all such costs for the chosen 
topics will not exceed the maximum permitted (4), but such that, at the 
same time, the sum of their values will be the highest possible for the 
data used. Algorithms are available to solve the dynamic programming 
problem* 

Advantages/Benefits ; The human judge will ordinarily not do as well as the 
application of dynamic programming. 

D isadvantages/Costs ; It is difficult to get the dynamic programming model to 
mirror real life examples. 

Resources Requires : None . 

Basic References ; 

Page, E. B. , Jarjoura, D. , & Konopka, C. D. Curriculum design through 

operations research. American Educational Research Journal , 1976, 13 , 
31-49 . 

Eck, R. D. Operations research for business . Belmont, CA: wadsworth, 
1976. 

Hillier, P. 3., & Lieberman, G. I. Introduction to operations research . San 
Prancisco: HOlden-Day, 1967. 



Truemah, R. E. Ah introduction to quantitative methods- in decision marking . 
New York: Holt, Rinehart and Winston, 1977. 
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Method : Queueing Theory 

Purpose ; Queueing Theory can be used to study waiting line problems 

Why arid When to Ose z Queueing Theory is used whenever there are delays in 
waiting lines* For example , students often wait in line at libraries* 
cafeterias, and showers; They also. wait in line at the offices of school 
nurses, vice-principals, and counselors* One of two errors will often be 
present: Either customers are waiting for service , or servers are waiting for 
customers* Most questions will center around three major questions: (1) How 
much of the time will the service channels (e.g* library checkout desk) be 
idle? {2) How many customers (e.g. _ students) will usually be waiting for 
service? (3) How long of a wait will each customer typically have? 



Basic Procedures ; Mathematical models are fitted to aspects of queueing 
behavior. If arrivals are assumed to be random, the probability that there 
will be no student arriving at the service channel is an exponential function 
of the number of minutes elapsed since the last arrival* One key distribution 
that follows from this is called the Poisson distribution* From this 
information it is possible to derive formulas which give the probability of 
finding a service channel idle at any particular time, the probability of 
having a certain number of customers either being served or waiting in line, 
the mean queue length, the expected number of customers waiting for service, 
the expected waiting time of a new arrival, and the probability of having a 
waiting line longer than a certain number of customers • 



ftdva ntag e s/B e ne f i ts ; Using the measured information of the number of students 
arriving per minute, and the number of students serviced in a minute, it is 
possible to_caicuiate_a_ wide, array Qf information about, the queueing system 
which is useful for making decisions about it; 

Disadvantages/Costs : _ within queueing theory there is no automatic answer; 
The theory does not lay out decision alternatives* For example, in the case 
of the library checkout desk, we.might calculate the waiting time and. decide 
that it is unacceptably large* Note that queueing theory does not help us 
make the decision of whether or not the waiting time is unacceptably long. 

Resources Required ; None • 

Basic References: 



Eck, R. D. Operations research for business . Chapter 8* Belmont, Cft: 
Wadsworth, 1976. 

Hillier, F. S., & Iiieberman', G* J* Introduction to operations research * 

Chapter 6* San Francisco: Hblden-Day* 1967. 

Page, E* B* Educational eva luat ibh-th rough-ope r afeibns-research * No. 3C of 
the Paper and Report Series of the Research on Evaluation Program. 
Portland, OR: Northwest Regional Educational Laboratory, 1979. 



Trueraah, R* E* An introduction to quantitative me thods— in— decision making . 
Chapter 8* New York: Holt, Rinehart and Winston, 1977* 
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Method ; Minimum-Maximum Goal Projection 
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Purpose ; The purpose is to show , the client the minimum and maximum that the 
evaluator can be expected to achieve. 



Why and When to Use ; in the process of planning an evaluation, the evaluator 
does a preliminary study in order to determine the feasibility of the 
evaluation. The question is whether the results to be expected from the 
evaluation wiH repay the resources that win have to be expended on it. 
Evaluatqrs specif y_modest minimum expectations which they were certain they 
could meet and. then_specify other expectations. which they might be able to 
meet if everything feU into place {e.g. , sampling could be accomplished in 
accordance with sampling principles; political intervention could be 
eliminated) • These minimum and maximum projections may be presented to the 
client. This would make it very clear to the client what he/she can expect 
from the evaluation and what he/she is getting for his/her money. The client 

may decide to increase the funds available to the evaluucor so that the 

minimum goals may be increased. On the other hand, the client might decide 
that he/she is not getting enough from the evaluation and thus negotiate with 
the evaluator the raising of the minimum projection. 

Basic-Pr oc e d ures ; In planning an evaluation, the evaluator makes projections 
of the minimum and maximum that can be achieved. 



Advan tages/B ene € its ; The evaluator and the client should have clear 
expectations of what can be expected from the evaluation. The evaluator 
cannot be accused of over-promises. ^ 

Oisadvahtages/Sosts ; The time and cost in carrying out. the, planning of an 
evaluation will be increased. It is difficult to make realistic projections 
of the minimum and maximum goals to be achieved. A 



Resources Required ; Extra time and cost in planning an evaluation. 

Basic References s * 

Guba, E. G. investigative journalism. In N. L. Smith (Ed.) , New 

techniques for evalaation ; Beverly Hills, Cft: SAGE Publications, 1981. 

Williams, P. N. Investigative reporting and editing . Englewood Cliffs, 
NJ; Prentice-Hall, 1978. 
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Method: Geocode Analysis 



Purpose : Geocode analysis is a technique developed in the field of geography 
for displaying and analyzing geographically related information. 

Why and When to Use ; Many evaluation studies concern the analysis of geo- 
graphic problems: the identification of individuals .needing specialized atten- 
tion in health, Welfare, arid education; the busing of children to achieve equal 
educational opportunity- Geocode analysis has been used in studies of school 
redisricting , the identification of Title I students, and in projections of 
student population growth- In evaluation, the technigue has been used to _plot 
differential reading growth due to remedial interventions at selected sites 

within a district- The technique is especially recommended for settings such 

as large school districts, community health service centers, and local law 
enforcement agencies which have pre-existing data files and geographically 
defined service districts. 

Basic Procedures : Geocode analysis uses the individual as the unit of analysis 
and aggregrates individual data over geographic areas, following are the 
steps involved: (I.) compile ah Address Coding Guide (e.g., address of each 
student)? (2.) builds an Individual Characteristics Data Pile (e.g. ± student, 
achievement scores); (3;) merge the Address Coding Guide with the Individual 
Char^ter istics Data Pile; and (4.) produce computer grids, plots, and 
contour maps of individual characteristics by geographic location. 

Advantages/Benefits : with geocode analysis one can examine geographically 
related data on thousands of individual cases. Computer-generated maps can be 
produced at various scales (several blocks, an entire school district, 
contiguous service areas, etc.) and at multiple time periods. 

D is advantages/Costs : Geocode analysis requires considerable manpower > money* 
and computer resources to develop and maintain an address coding guide, to 
merge the guide with existing individual records systems, and to produce 
computer -generated grids and maps* 

Resources -Required : Access to a computer and a computer program, clerical 
help is needed to compixe an Address Coding Guide and an Individual 
Characteristics Data Pile. 

Basic References : A detailed description of developing and maintaining 
digitized address coding guides, individual characteristics data files* and 
geographic data bases can be fcund in: 

Mclsaac* D. N* > Spuck* D. W. , & Stofflet, P. P. A users guide to a system of 
programs for the analysis of geographic areas . Madison, WI: Information 
Systems for Education* 1972. 

Spuck, D. W. Data— base -eonsiderat ions . Paper presented at the Annual Meeting 
of the American Educational Research Association, Chicago, Illinois, 
1972. (ERIC Number ED 064 894) 

For indepth discussions of geocode analysis and examples of applications, the 
reader is referred to: 

Costa, C. H. Application cf geocode -and mapping . Paper presented at the 

Annual Meeting of the_American_Educational Research Association, Chicago, 
Illinois, 1972. (ERIC Number ED 062 765) 

Spuek, D. W. Geocode analysis. In H. J, walberg (Ed.) Evaluating educational 
performance . Berkeley: McCutchan , 1978. 

For a general introduction to geocode analysis, see: 

Smith, N. L. Techniques for the analysis of geographic data in evaluation. 
Evaluation ^nd Program Planning , 1979, 2, 119-126. 



32 ^ 



Method Description Sheet 7 



Method : Trend Surface Analysis 

Purpose : Trend surface analysis is a technique from geology used tc generate 
three-dimensional contour maps on which to illustrate changes in important 
variables.. Variations in a particular variable oyer a geographic regions are 
partitioned into broad regional trends and small-scale local deviations from 
these trends* 

Why ana When to Use : Trend surface analysis has been used to study the 
relationship between geographic location and the following: school board 
elections and sources of local school support; the dissemination of 
i^f 9™ation on Title III projects? statewide educational heeds assessment; 
factors such as personal income, educational background, unemployment, number 
of dependents and financial support for local education* 



Basic Procedures : 1. Stations are the points in the geographic region chosen 
to represent a _lpcal ^geographic area. For example , in a study of fourth grade 
IQ scores,. each. school would be a station and the average IQ at thefourth 
2?^* I®?® 1 would be the station value * 2. Each station is identified by 
three data points: _ x, y coordinates which establish. the station's geographic 
location, and the station value for the variable of interest, the z 
coordinate* A .statistical npdelingprpc^ure is employed _ (e.g* , least squares 
polynomical trend fittings) and a surface equation is produced. _ 3. The 
surface equation can then be used to construct a contour map indicating 
regional trends and local trends* 4* Residuals can be computed and residual 
maps plotted to identify hidden trends and deviant stations* 5. Multiple 
surf ace maps can be overlayed to illustrate regional interactions between 
variables of interest* 

Advant ages/Benefits : Trend surface analysis is a useful technique for the 
visual display of large amounts of data; Using computers and high-speed 
plotters, the ev&luator can generate multiple data maps which portray 
geographic data relationships not discernible through tabular display* 

Disadvantages/Costs : The technique requires computer analysis and plotting 
facilities; 

Resources Required : A computer program and access to a computer with plotting 
facilities. 

Basic References : The following provide general introduction to trend surface 
analysis: 

Lewis, M. S. Trend surface analysis of community variables. Psychological 
Bulletin , 1977, 84, 940-949. 

Mclsaac, p. N. The application of -"trend surface analysis . Paper presented at 
the Annual Meeting of the American Educational Research Association, 
Chicago, Illinois, 1972. (ERIC Number ED 064 894) 

Mclsaac, D. N. The people _of the state: A description through trend surface 
analysis . 1973. (ERIC Number ED 082 673) 

Mclsaac, D. N. Trend surface analysis. In H. J. Walberg (Ed.) Evaluating 
educational performance . Berkeley: McCutchan, 1974. 

Smith, N* L. Techniques for the analysis of geographic data in evaluation. 
Evaluation and Program Planning , 1979, 2, 119-126. 
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Method : Social Area Analysis/Ecblbgicai Analysis 



Purpose ; Social Area i Analysis '_(Scolbgical Analysis) is a set of procedures 
which allow an evaluator to assess the relationships between an intervention 
program. or service delivery system and the demographic characteristics of a 
particular geographic region. 

Why and When— to-gse i Social Area Analysis would be appropriate for any 
evaluation where relationships between groups and geographic areas are a major 
concern, especially evaluations of education, health, and welfare service 
deli very. Sccial_ Area Analysis is_ npta discrete procedure* but a point of 
view. It is also a. collection of techniques used to study characteristics of 
Subpopulatibns within defined geographic areas, such as census tracts, 
counties, or service regions called catchment areas . Social Area Analysis 
employs such familiar techniques as factor analysis, cluster analysis, and 
multiple regression. 

Basic Procedures : 1. Define th« geographic region or catchment areas of 
interest and_ collect relevant demographic data. 2. Through factor analysis 
or cluster analysis, develop theoretically meaningful arid psychometrically 
stable indices of catchment area characteristics. 3. Through profile 
analysis, identify similar catchment areas. 4. Collect data on conditions, 
behaviors, or characteristics of interest, and study the relationships between 
these variables and catchment area indices through multiple regression. 5. 
Verify apparent relationships through direct inquiries of group members, 
exper im en tally controlled studies of treatment interventions, or time series 
designs. 

Advantages/Benefits ; Social Area Analysis requires no special data collection 
procedures except the collection of extensive demographic information which is 
usually available from census records* 

Pis advan t ages/Cos ta_ : Social Area Analysis involves considerable data 
collection and extensive statistical analysis. 

Resources Requir ed : Suitable computer programs with access to a computer. 

Basic References ; General introductions to Social Area Analysis are provided 
by: 

Kayr_?j? D. , Jr. Applications of social area analysis to program planning and 
evaluation. Evaluation and Prog ram Planning , 1978, 1} 65-78. 



Smith, N. itm Techniques for the analysis of geographic data in evaluation. 
Evaluation and Program Planning , 1979, Z, 119-126. 

Struening, E. L. Social areas analysis as a method of evaluation, in E. L. 
Struening & M. Guttentag (Ed3.), Handbook of evaluation research 
(Vol, 1.). Beverly Hi±l3, Cft: SAGE Publications, 1975. 

A presentation of the basic procedures is provided by: 

Shevfcy, E. , & EU?U, W. Social area analysis. Stanford: Stanford University 
Press, 1955. 

The most comprehensive statistical presentation can be found in: 

Tryon, R. C, & Bailey, D. E. cluster analysis. New York: McGraw-Hill, 1970. 
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Metho d: Concept Analysis 

P urpose ; Concept analysis helps the ^valuator clarify his thinking about 
general and abstract questions • Examples ares What i3 meant by compensatory 
education? What do we mean by effective reading instruction? What does it 
mean to say that a student knows something? 

Why and When To Use ; When the questions in an evaluation contain abstract 
terms, it is recommended that concept analysis be carried but. Concept 
analysis is also needed when individuals talk past each other using the same 
terms, or when different people interpret the same data differently. 



Basic Procedures ; There are three basic procedures— a generic-type analysis, 
a different istion-type analysis, a_nd_a conditions-type analysis. _A_ 
generic-type analysis deals with questions of the form* "What is X?* or "What 
is meant by X?" e.g./ "Whaf iii compensatory education?" or "What is meant by 
equality of educational opportunity?" In other words, "What are the basic 
features which make compensatory education a form of education and provide the 
criteria for distinguishing compensatory education from non-compensatory 
education?" Differentiation-type analysis is called for when the concept 

question in an evaluation requires making distinctions. For example, the 

question "What is the difference between teaching and indoctrination?" 
requires one to make distinctions between different forms of education. In 
differentiation-type analysis we clarify it and make a concept more useful by 
pointing to the different basic meanings it has. An example of a 
conditions- type analysis is: "Under what conditions would it be true to say 
that somebody knows something. The purpose of a_conditions-type analysis is 
to produce the set of necessary and sufficient conditions for the proper 
application of a concept to any of its many and varied instances. 

Advantages/Bene f its : Hfeips to clarify an ©valuator's thinking. Helps to 
answer concept questions on which depend on the answers to value, fact, and 
technical questions in an evaluation. Costs nothing except the evalua tor's 
time. 



Disadvantages/Costs : Difficult to carry out. Can be time consuming with 
little benefit to an evaluation. 

Resources Required : None. 



Basic References : 

Caulley, p. N. C oncept analysis in evaluation . No. 61 of the Paper and 
Report Series of the Research on Evaluation Program. Portland, OR: 
Northwest Regional Educational Laboratory, 1981 

Green, T. P. The acitivltes of teaching. New York: McGraw-Hill, 1971. 

Soltis, J. P. An introduction to the analysis of educational concepts. 
Reading, KA: Addision-Wesley> Second Edition, 1978. 

Wilson, J. Thinking with concepts. London: Cambridge University Press, 
1963. 
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Method: Thematic Matrix Analysis 

Purpose : _Tp identify^ tAeraes which characterize the program being evaluated. 
The themes help to describe and make the program more understandable to the 
audience. 

Why arid When to Use : When the evaluator wants to describe to the audience 
what goes on in a program and what its activities and processes are like. 
There are two types of situations in which such representations will be 
particularly useful. The first is when an interested audience is too distant 
from the program: members of the audience do not. know enough about it and 
carmbt experience it directly; under thesvs conditions they may feel 
uncomfortable about making judgments about the program or decisions about its 
future. The second is when participants in the program are too closely 
involved in its functioning to be able to step back from it and see it as a 
whole, yet different and unique to each participating observer. In both 
situations, there is a need for a thematic representation of the program that 
will allow its audience to experience something of the program, and perhaps to 
understand it better than before. 

Basic Procedures : Interview participants of the program* observe the .program, 
and analyze documents related to the program. Let the themes emerge from the 
data rather than preordinately beginning with themes. Make a matrix of themes 
crossed with instances of where they occur in the program. 

Advantage s/Bene fits : Themes help to organize and to make the data 
meaningful. Themes help to portray the realities of a program. 

Disadvantages/Costs : In order for themes to emerge, the evaluator must invest 
a considerable amount of his/her time in the evaluation. 

Resources Needed : None. 

Basic References : 

DeHa-Piaria, G. M. Literary and film criticism; In N. L. Smith (Ed.) , 
Metaphors for e valuati on . Beverly Hills: SAGE Publications, 1981. 



Kemmis, S. Telling- it-like-it -is : The problem of making a portrayal of an- 
educational program . Urbana, Illinois: Center for _ Institutional Research 
and Curriculum Evaluation, University of Illinois, 1974. 
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Method t Document Analysis 

Purpose ; Document analysis is the analysis of documents in order to gather 
facts. 

Why— and W h e n To - Use ; Document analysis is superior in finding out 
retrospective information about a program, and may be the only way that 

certain information is available, Document analysis is particuiarlyusefulat 

the beginning of an evaluation when the evaluator is trying to understand why 
the program is the way it is. Documents are an excellent source for 
^®termining the L purposes j .rationale, and history of a program. _ Dcing document 
analysis is usually a useful prelude to collecting new data; Knowing the 
purposes of a program helps the evaluator decide what data are to be 
collected. Doing document analysis one finds out when data have already been 
collected and what new data need to be collected. 

Basic Procedures ; Tracking is a process of working through documents looking 
for information that will confirm some hypothesis. Cdnteri t— Analy sis is the 
creation of categories in order to analyse documents.. C ase study aggregation 
is a tfeans for aggregating diverse case studies together Using a common 
conceptual framework so that the findings will Ui cumulative. 

Advart tag en /Bene f i ts : 1. Document analysis is superior to interviewing for 
collecting some kinds of retrospective data. 2. Information obtained from 
documents is often more credible than information obtained via observat on and 
interviewing. 3. Documents are convenient to use. 4. Documents are often 
available free or at little cost. 5. Documents are non-reactive: That is, 
it is not usual to find masking or sensitivity because the producer knows he 
or she is being studied by some social scientist. 6. Records save the time 
f^L^ohey _tA at original data coyTOtion requires. 7. _ Program .documents 
provide the evaluator with information about many things that cannot be 
observed because they may have taken place before the evaluation was begun and 
because they may include private interchanges to which the evaluator is not 
directly privy. 



Disadvantage s/Costs ; 1. Often> documents are written to make a program look 
good and thus can be misleading. 2. There is a dependency oh the memory of 
the person doing the reporting. 3. Although reams of material may be 
available, it often win not contain much of the needed information or be 

suff iciently detailed. 4. ^ __ I^uniehts may reflect clerical lapses, 

typographical errors, biases, or outright deception; 5. Agency records may 
be inaccurate, out of date, or months behind oh entries. 6. The definitions 
and categories used by an agency's records may be inappropriate for evaluation 
purposes; 7. Documentary facts never come "pure", since they are always 
refracted through the mind of the recorder. 8. Documents may provide 
unrepresentative samples. 

Resources Required ; None • 

Basic References i 

Caulley , D. N. Document analysis in program evaluation . No. 60 of the Paper 
and Report Series of the Research on Evaluation Program. Portland, OR; 
Northwest Regional Educational Laboratory, 1381. 

Guba, E. G. Investigative reporting. _ In N. L. Smith (Ed.), Metaphors for 
evaluation; Sources of new methods. Beverly Hills: SAGE Publications, 
1981. 

Guba, E. G. , arid Lincoln, Y. 5. Effective- evaluation. San Francisco: 
Jcssey-Bass, 1981. 

Murphy, J. T. Getting the facts . Santa Monica: Goodyear , 1980. 



37 



Method "Description Sheet 12 



Method : Legislative History 

Purposes A legislative history is constructed to determine the true intent of 
legislators regarding a law enacted by the legislative branch of a government. 

Why and When to Use ; Legislative histories can be cione with those evaluative 
studies which are based on programs. haying a history in the legislature. ESEA 
Title I, Title IV, and the Title I Technical Assistance Centers are examples 
of programs which have theix dir^t soik 

asked include: "Are the aims and objectives of the program the same as those 
intended by the legislature?" and "Are the outcomes of the program the same as 
those intended by the legislature?" Such questions are significant when the 
legislators are an important audience of the evaluation, 

Basic Procedures ; The essential steps in doing a legislative history are as 
follows: 

1. Identify critical sources of official information. 

2. Get the bill, congress, and statute number for the particular subject 
under examination. 

3* Find and examine both House and Senate Committee reports, as well as 
any conference reports* 



4; Check the proceedings of each house and read the debates therein. 



5; Examine the hearings and any other allied material* 

Advantages/Benefits : By doing. a legislative history* the evaluator is able to 
learn the true intent of legislators regarding a program which has been 
mandated by the legislature and which is being evaluated. By knowing the 
legislators 1 intent the evaluator, can be more responsive to the legislators' 
information needs. No direct costs are involved. 

Disadvantages /Costs : Difficult for the evaluator to find his/her way around 
legislative documents. Time consuiTi'.? :? carry out. Documents recording 
legislative intent are not as extensive for the state legislature as for the 
federal legislature. 

Resources Required : Access to a library that has legislative documents. The 
acquaintence of a .government document librarian should be sought to help the 
evaluator find his/her way around documents. 



Basic References : 

Caulley, D. N. Legislative history and evaluation . _No. 62 of_ the Paper _ 
and Report Series of the Research oh Evaluation Program. Portland, OR: 
Northwest Regional Educational Laboratory, 1981. 

Folsoh, G. B. Legislative history . Charlottesville: University Press 
of Virginia, 1972. 
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Method t The Key interview 

Pursose : The key interview has a variety of purposes: to confront the 
interviewee with the results of the evaluation to that point; to elicit the 
interviewee's response i to allegations about the. program being evaluated? to 
test the evaluative data for accuracy? to get the interviewee's side of the 
story; and to get new information that is available only from this 
knowledgeable source* 

Why and When to Uses The key interview is appropriate when evaluators find 
themselves in situations in which a key informant (for instance, a project 
director) is hostile and uncooperative, as, for example, in a third-party 
evaluation in which the informant feels threatened. 

The key interview is_ also appropriate when the evaluator wants to test the 
first dra*t of an evaluation report for the following: to determine its 
credibility to various audiences, to solicit their reactions, and to get 
further details that may help clarify issues being studied. 

Basic Procedures : The key interview typically involves at least the following 
steps. 

_W fining entree. Some audiences may refuse to see the evaluator; others 
may argue passively; still others may actively seek but the evaluator to 
present their side of the story 

(2) Preparing. The evaluator (and at least one other person who will, 
accompany the evaluator) should prepare by rereading the entire accumulated 
file of material, formatting the interview, and writing and arranging the 
questions that will be asked. 

(3) Beginning the interview. The interest of the evaluators in finding out 
the facts should be stressed. The questioning should begin with easier 
questions that are not likely to be emotionally tinged. 

(4) Controlling the interview. The evaluator should establish and maintain 
control of the interview throughout. 

(5) Confronting the interviewed. Eventually the interviewed should be 
confronted vith the evaluation findings that have beei? developed so far. 

(6) Recording. Everything that transpires at the interview should be 
recorded. 

Advantages/Benefits : To confirm evaluative facts and interpretations, and to 
obtain crucial new information. 

Disadvantages /Costs : May alienate key audiences to an evaluation. 
Resources Required : Two evaluators and a tape recorder. 



Cuba, E. G. Investigative journalism. In N. L. Smith (Ed.), New techniques 
for evaluation. BeverAy Hills, CA: SAGE Publications, 1981. 

Williams, P. N. Investigative reporting and editing . Englewood Cliffs, NJ: 
Prentice-Hall, 1978. 
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Method j interviewing: Circling, Shuffling and Pilling 

Purpose t The terms; "circling," "shuf fling," and "filling," are three 

techniques for extending and rounding out information. 

Why and When to Ose ; When the evaluator has incomplete information on a topic 
and want3 to use interview data to extend and round oqt that information, then 
circling, shuffling and filling are appropriate techniques. 

Basic Procedures i Circling involves taking information obtained from one 
source and running it back around the evaluator 1 s ..circle of contacts for 
refutation or confirmation. The contacts are assumed to be cooperative. 

Shuffling is like !_circiing but with two differences: the contacts are not 
assumed to be cooperative, and the evaluator makes an effort not only to 
confirm or deny but also to bu ild upon the original bit of information. For 
example, when person. B is presented with the tidbit of information gleaned 
from person A and is asked to confirm or deny it, he is also pressed for more 
details, if he denies the information, he is asked what is really true, what 
really did happen. If he confirms the information; he is asked to supply more 
details. Whatever person i_B supplies ( and, of course, whatever person A had 
supplied) are then presented to person C, and so on round the chain. 



Pilling involves two things: building _up_ a perimeter around the investigative 
area— a boundary-— and then working to fill in the gaps that exist within it. 

Advantages/Benefits ; To confirm or disconfirm evaluative facts and 
interpretations, and to obtain crucial new information 

Disadvantag e s/Costs ; Circling, shuffling and filling may be very time 
consuming • 

Resourc e s Requir e d ! None. 

Guba, B. G. Investigative journalism. In N. L. Smith (Ed.), New techniques 
for evaluation . Beverly Hills: SAGE Publications, 1981. 



Williams, P. N. Investigative reporting and editing . Englewood Cliffs, NJ: 
Prentice-Hall, 1978. 
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Method ; Documentary Files and Summaries 

Purpose ; Files and summaries are an important permanent record and s.*rve as 
the basis for the evaluation as the e valuator writes it. 



Why and When to Use ; Files and summaries have a great deal more utility than 
simply as a convenience in cataloging collected material and serving as a 
memory aj.d. They are, in fact, an important tactical investigatory tool in 
their own right. 



Basic Procedures ; 
Stage 




Hypotheses to be Checked and Gaps to be Closed 



1. Filing begins with folders for each of the individuals involved in the 
evaluation (Stage 1). 



2. As the persons originally involved enter into transactions with others (a 
transaction may be as simple as a meeting for lunch) , folders are 
developed related to each transaction (Stage 2). 

3* Simultaneously, profiles are developed on each of the persons for whom 
files have been established* 

4. Chronologies are developed for the various transactions (Stage 3) • 

5. As the files develop, important items are cross-referenced* 

6* Each file is also summarized on a systematic basis, so that the evaluator 
need not deal with the. bulky original items each time he enters the file, 
but rather with a summary of what it contains. 

7. Entries in the several files will give ri3e to hypotheses to be checked 
or will point to informational gaps that probably ought to be closed 
(Stage 4). 

Advantages/Benef its ; Files and sunm^ies ^e valuable in ordering, and 
keeping. track of, a large amount of information. The approach is especially 
fruitful in the monitoring of a program. 

Pis advantages /Cost : Maintaining and keeping files can be very tedious and 
timeconsuming • 

Resources^ 4tequlred s Facilities for filing. 
Basic Reference: 



Guba, E. G. Metaphor adaptation reports Investigative journalism . 

No. 4 of the Paper and Keppr t Series of the Research on Evaluation 
Program, Portland, OR: Northwest Regional Educational Laboratory, 1978. 
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Method : Cost-Benefit Analysis 

Pappose: to compare the costs and benefits to society of various policy 

alternatives 

When and Why to Ose r Cost-benefit analysis can be used whenever all the 
benefits can be readily converted into pecuniary values or when those. that 
cannot be converted tend to be unimportant or can be shown to be similar among 
the alternatives that are being considered* 

Basi ^Procedures : For each policy alternative, estimate the costs and convert 
the benefits into pecuniary terms* For each alternative calculate a 
cost-benefit ratio and select the alternative with the lowest cost-benefit 
ratio* 

Advanta^es7^ene€its: The advantage of obst-benef it analysis is the range of 
comparisons that can be made among alternatives , not only within a particular 
service area , but_araong them. Thus; educational projects can be ranked 
according to their cost-benefit ratios> but they can also be compared with 
health and transportation projects that have been evaluated for their costs 
and benefits* 

Pi sad vantage 8 /Costs: The disadvantage is that benefits and costs must be 
assessed in pecuniary terms. It is not often possible to do this 
systematically and rigorously. For example , while the gains in earnings and 
certain self-provided services attributed to higher levels of literacy (say) 
might be assessed according to their pecuniary worth, how does one assess such 
benefits as improvement in self-esteem of the newly-literate adult populations 
or their enhanced appreciation of reading materials? 

Re sou r ce 3 Needed : Hone • 

Basic References ; 

Rothenberg, J. Cost-benefit analysis: A methodological exposition. In M, 
Gut ten tag and E. L. Struening (Eds*3, Handb ook of o va luation research # 
Vol. 2. Beverly Hills: SAGE Publications, 1975; 

Levin, H. M. Casebook -on cost analysis -in- educa^ional-evaiuation (No* 33 of 
the Paper and Report Series of the Research on Evaluation Program) • 
Portland, ORi Northwest Regional Educational Laboratory, 1979* 
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Method : Cost-Effectiveness Analysis 

gggpooe; To compare the costs and benefits to society of various policy 

alternatives 

When and Why -to-gse ; There are many evaluative situations in which it is 
possible to assess the educational results of particular alternatives while 
not being abletoplace pecuniary values on those. results. For example, it is 
difficult to conceive of acceptable methods for placing pecuniary worth on 
test /*?9*~?_ 9ains _in arithmetic .operations or on improvements in self 
confidence. _In_ these cases, it seems more appropriate- to compare the results 
of programs in terms of their effectiveness bh some criterion or set of 
criteria, then comparing these measures of effectivenes with the costs of the 
alternative programs. 

Baeie-Pgeeedures: For each policy alternative, estimate the costs and measure 
the effectiveness in terms common to all alternatives. For each alternative, 
calculate a cost-effectiveness ratio and select the alternative with the 
lowest ratio. 

Advantages/Benefits; It is commonly found that the most "eff ectlve" approach 
is not always the most cost-effective. Yet, without an analysis of costs it 
will not be possible, to know_ this. Further the adoption of the most 
"effective" alternative can actually cost many times as much as the most 

oost-effective one. Cost^effectiveness comparisons require only that the 

impact of alternative strategies, along with their respective costs, be 
derived, while cost-benefit analysis requires that we put a monetary value on 
the impact. The results of policy-oriented experiments or quasi-experiments 
lend themselves naturally to cost-effectiveness comparisons. 

Disadvantages/Costs t Cost-effectiveness is of no Value for selecting among 
alternatives that address different objectives. Measuring the effectiveness 
of programs requires the setting of experimental or quasi-experimental 
designs, whicti is difficult and time consuming. 

Resources Required ! None . 

B asi c R eferences 

Levin, H. Cost-effectiveness analysis in evaluation research. In M. 

Guttentag and E. L. Stf ueriirig, (Eds.) , Handbook of e valuation research . 
Vol. 2. Beverly Hills: SAGE Publications, 1975. 

Levin, H.M. Casebook on coat analysis in educational evaluation No. 33 of 
tha Paper and Report Series of the_Research on_ Evaluation program). 
Portland, OR: Northwest Regional Educational Laboratory, 1979. 
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Method : Cost-Utility Analysis 

P urpos e ; To compare the costs and benefits to society of various policy 

alternatives 

When and Why to Use ; When benefits cannot be_ measured in monetary terms or in 
terms of effectiveness* cost utility is the alternative. When subjective 
assessment must be made about the nature and probability of educational 
benefits as well as their relative worth, cost utility analysis may be an 
appropriate tool. 

Basic Procedures : First, the decision maker uses the information available to 
him or her to ascertain the probability of achieving particular educational 
outcomes with each of the policy alternatives. Uith the cost-utility approach 
there can be more than one outcome per alternative. Second, the decision 
maker places relative values on each of the educational outcomes to weight 
them according to their desirability. The method for doing this is to rate 
each potential outcome on a _ scale of utility which reflects the desirability 
of that outcome. For example, a decision maker could assess the value of each 
outcome on a 0-10 point scale with egual_ intervals , _in _ which 10 represents the 
highest value. Third, the subjective probability of each outcome is 
multiplied by the utility placed upon the outcome and adding these products 

across outcomes. This calculation is done separately for each policy 

alternative- Finally, costs are divided by the expected utilities to obtain 
cost utility ratios for each alternative. 

Advan tage s /Benef its ; Cost-utility analysis permits the use of a wide range of 
types of both qualitative and quantitative data in forming the decision. The 
advantages pf the cost-utility, approach are that the data requirements are 
less stringent than cost-benefit or cost-effectiveness approaches, that a 
large number of potential outcomes can be included in the evaluation, and that 
imperfect information and uncertainty can be addressed systematically . 

Disadvantages/Costs: The highly. subjective nature of the assessments of 
effectiveness and the values placed upon them. by_the decision, maker prevents 
the kind of replicate ility from analysis-to-analysis that might be obtained 
with the more stringent cost-benefit and cost-effectiveness approaches. 

Resources Required ; None. 
Basic References ; 

Edwards, W. , Guttentag, M. , a_Snapper, K» A deciaion-Uieocetic approach to 
evaluation research* In E. L. Struening and M. Guttentag, (Eds.), 
Handboo k of -eva luation research , Vol. 1. Beverly Hills; SAGE 
Publications, 1975. 

Stokey, E., & Zeckhauser, R. A primer for policy analysis . New York; W. W. 
Norton & Co., 1978. 

Levin, H. M. Casebook on cost analysis in educational evaluation (No. 33 of 
the Paper and Report Series of the_Research on_ Evaluation Program). 
Portland, OR; Northwest Regional Educational Laboratory, 1979. 
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Method ; Cost-Pea si biiity Analysis 

Purpose : to compare the costs to society of various policy alternatives 

When and Why to Us e ; Cost- feasibility analysis represents a method of 
estimating the costs of _an alternative in order to ascertain whether it can be 
considered. That is, if the cost of any alternative exceeds the budget and 
other .available resources, there is no point in doing any further analysis. 
As a concrete illustration, one might view the situation of compensatory 

education in which a specified amount is available for augmenting the 

education of each disadvantaged child. If this amount is $400 per Child, thc?n 
any alternative that exceeds this constraint would not be feasible. 

Basic Procedures : The cost of every policy alternative is determined. 

Advantages/Benefits : Cost-feasibility analysis can determine whether it is 
necessary to do the following: estimate the benefits of a cost-benefit 
analysis, determine the effectiveness of a cost-effectiveness analysis, or 
estimate the utilities of a cost-utility analysis. 

Disadvantages/Costs ; Cost-feasibility represents a limited form of analysis 
which can only determine whether alternatives are within the boundaries of 
consideration. It cannot be used to determine which ones should actually be 
selected. 

Resources Required : None. 



Ba sic Re ferences : 

Levin, Hi Mi Casebook on cost ahalys is^n-ed^ateibna^-ev^iiia^ion (No. 33 of 
the Paper and Report Series of the Research on Evaluation Program) . 
Portland, OR: Northwest Regional Educational Laboratory, 1979. 



45 



Method- D e scripti on ^heet_2Q 



Method ; Blanket Sampling 

Par pose ; To cover as much of the entire span of events as is possible 

Why. and When to Use ; Blanket sampling is used when the sampler wants to cover 
a wide range of events* 

Basic Procedures ; The sampler moves about freely, and fairly constantly, from 
location to location so as to cover a wide range of events. 

Advantages/Benefits ; This method of sampling does hot constrain the sampler 
to follow a given session or event through to completion. 

Disadvantages /Coat s ; If photography is being used to blanket sample, the 
photographer will gather a .rather large proportion of liminal events, 
generally at the expense of sequences showing development of action, changes 
o*_3?pups r shif ts_in_proxemicsj etc. .Basically, _this_ is themethodpiogy pf_ 
traditional photojournalism, and the fact that it tends to gather only liminal 
events probably _accounts_ for both the strong impact and the uncontrolled 
biases of most photographic documentaries. 



Resources Required ; None* except if using photography; then camera equipment 
will be needed by the evaluator. 

Basic References; 



Templin> P. A. Photography as an evaluation - technique (No. 32 in ^he Paper 
and Report Series of the Research on Evaluation Program) . Portland, OR: 
Northwest Regional Educational Laboratory > 1979. 

Templin, P. A. Handbook in evaluating with photography (No . 63 in the Paper 
and Report Series of the Research on Evaluation Program). Portland, OR; 
Northwest Regional Educational Laboratory, 1981. 



d , 
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Method : Shadow Sampling 

Purpose : To carry out detailed observations of how an individual or 

individuals spend their time 

Why and When to Use : Shadow sampling is used when it is desired to find out 
what activities the participants of a program engage in. 

Bas ic Pr ocedures : Shadow sampling involves following a single subject through 
a program and recording the experiences and interaction of that individual. 

Advanteaes/Benef its : The data collected will all share a common independent 
variable: the individual being shadowed. 

Disadvantages/Costs s The individual chosen for shadowing may introduce 
systematic bias into the data through the. types of interactions and situations 
in which she or he typically and habitually engages (or does hot engage)* 
Taking this viewpoint, shadow sampling may result in collecting data about the 
individual, rather than about the program. This problem cart be over come, to 
some ex ten t , by s hadowi ng several sub j ec ts , e i ther s imultaneou sly or 
consecutively. However simultaneous shadowing generally requires multiple 
observers, which is likely to be very obtrusive, while consecutively shadowing 
several subjects, through a program of any length may require unrealistic 
amounts of contact time. 



Resources Required : One or more observers and time to carry out the 
observations 

Basic References : 

Templin, P. A. Photography as an evaluation technique (No. 32 in the Paper 
and Report Series of the Research on Evaluation Program). Portland, OR: 
Northwest Regional Educational Laboratory, 1979. 

Templin, P. A. Handbook in evaluating with photography (No. 63 in the Paper 
and Report Series, of tiie Research on Evaluation Program). Portland, OR: 
Northwest Regional Educational Laboratory, 1981. 
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Methodi Time-Baaed Sampling 

Purpose: Time-based sampling reveals shifting patterns of persons in a fixed 
space over time. 

Why and When to Dse: Time-based sampling can i be ( extremely useful in studies 
of the use of space and can also provide information on the changing relations 
between groups and between members of groups. 



5a sic Procedures s The spatial coverage (e.g.r a room) is fixed and samples 
are taken at regular intervals _of _ time- One method of collecting data with 
this method is to use a 16mm movie camera fitted with an extreme wide-angle 
lans (often .covering a .whole room) , and an intervaloraeter . .Alternatively f a 
35mm still camera can be used, fitted with a bulk film magazine, motor drive, 
extreme wide-angle lens, arid intervalometer . The advantage of the movie 
camera is that the resulting film can be projected as a time-lapse movie f thus 
showing motion patterns aha changes in the spatial distribution of persons in 
the environment* Oh the other hand* time-based photo sequences taken *ith the 
35mm still camera can be examined minutely for such details as eye contact , 
and prints can be measured to quantify the proximity of subjects K> one 
another. 



Adv antag e s /Benefits ; Time-based sampling is easy to set up and carry out. 



D isadvantages/Costs : Collecting data oh a constant time basis may tend to 
exaggerate, the _ importance of the large proportion of "inactive" time that 
tends to separate human events. 

Resources Required ; None> except if using photography;, then camera equipment 
will be needed by the e valuator. 



Basic References; 



Templin, P. A. Photography as _an__ evaluation technique (No. 32 in the Paper 
and Report Series of the Research oh Evaluation Program). Portland, OR: 
Northwest Regional Educational Laboratory* 1979. 

Templin f P. A. Handbook in evaluating with photography (No. 63 in the Paper 
and Report Series of the Research on Evaluation Program). Portland, OR: 
Northwest Regional Educational Laboratory , 1981. 



ERLC 
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Method x 



Event-Based Sampling 



Purpose ; 



To sample as many events of a certain type as is possible 



Whyand When to Use ; Event-based sampling is used when the research question 
posed for the study is concerned not with a program or system as a whole, but 
with a particular type of action, interaction, sequence, or event within the 
overall context. Events may be relatively broad ("question-and-answer 
periods," "mealtime groupings"), or very specific ("handshakes between persons 
of the opposite sex after the presentation"). 

?- * *P?eau r e fM _ _ Tne sampler selects a particular type of event and samples 
the occurrence of it. 



D i 3adva n tagea /Cos t3 ; _Eyent-based sainpiing_is generally only appropriate in 
response to_ specific research questions. It should probably be accompanied by 
some more global technique (e.g., blanketing the event area, having another 
person take field notes, tape recording) to provide contextual information for 
interpreting the category of specific events. 

Resources Required ; None, except if using photography; then camera equipment 
will be needed by the e valuator. 

Basic- References ; 

Tempi in , P. A. Photography as an evaluation technique (No . 32 in the Paper 
end Report Series of the Research on Evaluation Program) • Portland, OR: 
Northwest Regional Educational laboratory, 1979. 

Teraplin,_P. A« _ Handbook in evaluating with photography (No. 63 in the Paper 
and Report Series of the Research oh Evaluation Program) • Portland, OR: 
Northwest Regional Educational Laboratory, 19S1; 




fits : Event-based sampling is easy to set up and carry out. 



ERLC 
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Method ; Dimensionally Based Sampling 

Purpose ; To collect data from the widest possible range of types of 

situations 

Why ah d-ffheh- to-Use ; Dimensionally based sampling would appear to be useful 
where the initial aim of the study is .descriptive, appraisal* but the 
constraints of time, space, and work force make it impossible to cover all 
events in the program. 

Basic Procedures ; Any information available about the program being evaluated 
(field notes, previous reports, informants reports, printed programs, 
informational brochures, etc •), is examined ^determine a set of apparent 
dimensions for classifying the events and components of the program. _ For 
example, in evaluating a conference, the evaluatOr used the dimensions of 
subject matter of .sessions and raeeting_format. The subject matter of sessions 
was divided into "theoretical" versus "applied. " The meeting formats were 
categorized intp:_ _a single long presentation, a series of topically related 
presentations, panel discussions, workshops, etc. The categories of subject 
matter were cross-indexed with the categories of meeting format, and sampling 
was carried out across the resultant cells* Events are then selected to 
represent the largest possible range of dimensions and combination of 
dimensions. Other constraints, such as observing . a whole event , or spending a 
predetermined amount of time at each event, may also be imposed. 

Advantages /Bene f its ; The advantage of the sampling procedure is that it__ 
offers some guarantee of representativeness in the description of the program 
being evaluated. 

Disadvantages/Costs : Dimensions are chosen at the beginning of the evaluation 
bat wiH usually need to be progressively modified during the course of the 
study to reflect the specific research questions that arise as a result of the 
analysis of the data. 

R e sourc es Regu ic ed s None, except if using photography, then camera equipment 
will be needed by the e valuator. 

Basie-Ref erehees : 

Tempi in, P. A. Photography as an evaluation technique (No. 32 in the Paper 
and Report Series of the Research on Evaluation Program) . Portland , OR: 
Northwest Regional Educational Laboratory, 1979. 

Templin, P. A. Handbook in eval gating with photography (No. 63 in the Paper 
and Report Series of the Research on Evaluation Program). Portland, OR: 
Northwest Regional Educational Laboratory, 1981. 
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Method ; Research Briefs 

Pwpeae-; . A research brief is a written and condensed statement that gives 
important information about research* 

Why and When to Use ; A research brief is used when part of the expected 
audience to an evaluation report will not read the complete report but will 
read a brief outline. 

Basic Procedures : Report summary ; One of the best known research briefs is 
probably the summary that appears at the end of most, if not all, final 
research reports. 

Executive summary : A variation of the report summary, the executive 
summary is placed at the beginning of the report. 

Memos.: One simple application is the attachment of a single-page memo to 
the top of the research report as a means of capturing audience attention and 
introducing the report. _ 

Cockles: A google is a one-liner designed to make its owner appear 
intelligent and well read. An ever moderate supply of googies should enable 
one to speak intelligently _aboat any of several evaluation studies. 

Embedded quotations ; This is either an abridgment or a direct restatement 
of a portion of a research report and is set apart within the text of the 
report. 

News items: The news item is similar to the report summary but appears in 

newspapers or newletters. 

abstracts: This is a very concise, accurate and informative report 
summary • 

Advantage s/Benefits : The research brief helps the reader to grasp the 
essentials of an evaluation report without the necessity of investing the time 
and effort reading the whole report. 

Disadvantages/Costs : The audience of the brief could be misled through 
incomplete information or insufficient qualification. 

Resources Required : None. 
Basic Reference : 

Macy, D.Ji Research briefs, in N.t* Smith (Ed.), Communication strategies in 
evaluation. Beverly Hills, CA: SAGE Publications, 1982. 
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Method: 



Appeals Procedures 



Purpose ; 



To review evaluation findings to determine whether they 



adequately support the claims made 

Why and When to Use : After completing a draft evaluation report, the 
evslaator can share the draft with the program staff to. verify the factual 
accuracy of the report's content arid interpretations before finalizing the 
report For large summative evaluations, it would be prudent to have the 
procedures reviewed systematically by an independent party ^ and an independent 
lodgment could be made as to whether the conclusions reached by the evaluator 
are warranted v y the supporting data. 

Bas4e-Proeedures : Set up ah appeals grocer a ^whichthe content and findings 
of an evaluation report may be questioned by audiences to the evaluations. 

Advantages/Benefits : If audiences feel they have been misrepresented in an 
evaluation reports or if inappropriate instruments or data analysis procedures 
were used, the audience can have recourse through an appeals process. 

Disadvantages/Costs : The appeals procedures may allow the program staff to 
delete legitimate references to program weaknesses or substantiated 
interpretations. 

Resources Required : None • 

Basic References : 

Green , M. Lega 1_ procedure «_ Collier's encyclopedia . Chicago: Collier 
Publishing, 1965, 14, 438-453. 

Owens, T. R. & Owen, T. R. ,_ taw. N. L. Smith (Ed.)> Metaphors for 
evaluation. Beverly Hills: SAGE Publications, 1981. 

Sobeloff, S., & Reitze, C. Standards -relating to criminal appeals: American 
Bar Association Projec t on Minimum Sta n da rds— for- Criminal- Jus tiee . 
Tentative Draft. New York: Institute of Judicial Administration, 1969. 
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Method t Storytelling 

Purpos e : To create a .believable and coherent story from the myriad 

educational events and activities that take place in an educational setting 
during a specified period of time 

Why and Wh e n t o— Use : Creating a story of an evaluation program occurs not 

only at the end of_the evaluation inquiry when the evaluation report is 
w ?*5?? n ^ 5? t also from the first moments Of the evaluation effort. The 
emphasizing of particular facts and events represents a fundamental 
structuring of information that is basic to storytelling. 

Basic Procedures : A structure or plot is devised to enable the evaluator to 
choose what information .should be collected, how it shpuid_be_cbilected f and 
from whom it should be collected. Grapple with the question of purpose. Why 
is the story (evaluation) being told? To what end will the story Revaluation) 
be_used? , Choose what information should be collected. Identify those 
essential components , the major events and existents, that are necessary for 
thtt development of the plot. Decide from whom information should be collected 
and how it should be collected. 

Focus the story (evaluation) by relating all of _ the events and existents that 
were investigated during the conduct of the evaluation (story) to each other 
and to the overall purpose of the inquiry. This will enable the evaluator to 
monitor whether they "made sense" in relation to each other and whether they 
add to the unity and coherence of the investigation as a whole. 

advantages/Be ne f i ts : Devising a plot helps the evaluator to choose what 
information should be collected , how it should be collected , and from whom it 
should be collected. 

Disadvantages/Costs ; Creating a plot at the beginning of an evaluation does 
not allow new facts and issues to emerge as the evaluation progresses. 

Resources Required; None; 



Basic— References ; 

Wachtmah, E. I*. Evaluating and. story telling: The narrative quality and 

structure of educational evaluation. In N. L. Smith (Ed.), Communicating 
in evaluation; Alternative forms of representation. Beverly Hills: SAGE 
Publications, 1982. 
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M e thod ; Compelling the Eye 

Purpose: To compose (design) an evaluation so that the mind of the decision 
maker is compelled to deal with the minor decisions, bat in a way that major 
attention is always turned back to the major decision. 

Why and When to Use ; A request for an evaluation is typically an indication 

that a major decision is to be made. Just _a3 typically A _a_nuraber of minor 

decisions must be made. This major and minor decision composite is analogous 
to the artist's major point of interest and minor interest points. 



Basic Procedures ; As the artist i3 concerned with compelling the eye, the 
evaluator is concerned with "compelling the mind." If the evaluation is 
designed so that it attends only to the major decision, the evaluator has 
created a quick exit avenue from the evaluation problem: thus it becomes an 
evaluation flavored by its failure to deal with the total evaluation problem. 
If _ the evaluation -is designed so that too much energy is expended on any or 
all of the minor decisions, or if it doesn't compel the decision maker to 
return to the major decision, it is again a flavored evaluation. 

Advantages/Benefits; There is a balanced approach in attending to major and 
minor decisions. 

^isadvM%a^es/Gogt^ : The writer of the evaluation report must be skillful in 
order to produce a report that "compels the mind." 



Resources Requlrad : None . 
Basic- References : 



Gephart, W. J. Watercolor painting. In N. L. Smith (Ed.) , Metaphors for 
evaluation . Beverly Hills: SAGE Publications, 1981. 

Kautzky, T. The Ted Kautzky pencil book . New York: Van Nostrand Reinhold, 
1979. 
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Method : Representation of Reality 

Purpos e ; To write evaluation reports that represent the reality of the 
prog rams being evaluated. 

Why and When -to Use : The technique can be used for reports of evaluations of 
programs when it is desired to give the reader a vivid representation of 
reality. 

Basic Pr oc edures ; The device used to represent reality is the use of a 
"_3tream_pf .consciousness" or "criterion monologue. " In representing reality 
we are given not just one person whose consciousness is tendered but many 
persons, with frequent shifts from one to the other. Reality (time passed] is 
violated to get at reality (the conscious wanderings of the mind). The 
treatment of time is the characteristic and distinctive feature of the 
technique. Reality is represented by the interior monologues of the 
characters of the story. The time it takes for such interior monologues is 
certainly, in reality, not as long as the time it takes to read or hear them. 
The exterior reality of the moment is nothing but an occasion to release 
things seen only by reflection in consciousness and not tied to the present or 
the occurrence which releases them. There are three characteristics of this 
representation of reality. 

Most objective facts appear as reflections in the consciousness of 
several characters. 

(2) The reality of time passed in exterior events. is_ violated by devoting 
more time to get at the reality of conscious wanderings of the mind. 

(3) The things that happen to a few individuals in a random moment 
represented in completeness, exploited fully, is favored because it reveals 
more of the elementary things our lives have in common than the course of a 
human life over a prolonged time, arbitrarily pruned and isolated. 

Advantag es/Benefits ; The technique gives an added dimension of understanding 
bo the reader of an evaluation report. 

Disadvantages/Coats: It takes considerable writing skill to portray reality. 
Resources Required ; None. 
Basic References ; 

Della-Piana, G. M. Literary and film criticism. In N. L. Smith (Ed.), 
Metaphors for evaluation . Beverly Hills; SAGE Publications, 1981 

Auerbach, E. Memesis^ — The representation of reality in Western literature . 
Princeton; Princeton University Press, 1953. 
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Method : accurate, Sharp Descriptions 

Purpose: To write evaluation reports that involve accurate, sharp, loving 
descriptions. 

Why and When to Use s This technique can be used for evaluation _ reports of 
programs that are regarded by the evaluator as meritorious in some way. 

Basic Procedures : Write up the evaluation report as an accurate, sharp, 
loving description. 

Ad van ta g e 3 /Benefits : Accurate, sharp, loving description of a program makes 
the program come alive for the reader. The reader gains a vivid understanding 
of how the program operates. 

Disadvantages/Costs : It takes considerable writing skill to write an 
accurate f sharp , loving description. 

Resources Required ; None. 

Basic References ; 

Della-Piana, G. M. Literary and film criticism. In N. L. Smith (Ed.) , 

Metaphors for evaluation. Beverly Hills: SAGE publications, 1981. 

Jarrell, R. - The third book of criticism . New York: Farrar, Straus, 
Giroux, 1965. 
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Method : Graphic Displays 

Purpose s To get the people in the target group to respond to a visual 

^ a 9e_which will communicate a message in a manner the group can readily 
understand • 

Why and When to Use : Graphic displays can be used to clarify complex data, to 
emphasize relative magnitudes, to help identify trends, and to demonstrate 
relationships. 

Basic Procedures ; The first step toward using the techniques of graphic 
display to improve the reporting of evaluation results is to incorporate them 
appropriately^ into the body of our traditional printed evaluation reports in 
the form of well-designed illustrations, graphs, charts and so on. The next 
step is to supplement these written reports by other forms of presentation 
Sc ^°^^^_ ^^he_ needs of the audiences and as the availability of resources 
Permit. Examples of ways to present the results of evaluations that lend 
themselves readily to graphic display techniques are exhibits, posters, 
brochures, overhead tranparencies, slide/ tape presentations, videotapes, and 
films. 

Advantages/Benefits ; Graphic displays help the audience understand the 
evaluative information that we present. 



Disadvantages/Costs t One difficulty with the use of graphic techniques is 
that they require more time and money for both report planning and execution. 

Resources Required : Access to a graphic designer. 

Basic-Reference : 

Hathaway, W.B, GrapWc display procedures. In N.L. Smith (Ed.), 

Communication strategies in evaluation . Beverly Hills, CA: SAGE 
Publications, 1982. 
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Method; The Stem and Leaf Displays and Box Plots 



Purpose ; To display a batch of data so that its properties are visually 
discernible. 

Why and When to Use ; The stem and leaf display provides a visual display of 
the shape of a distribution similar to that provided Hy a histogram. Box plots 
are used when the evaluator wishes to explore the properties of his/her data 
and when he/she wants to give the reader of the evaluation report a visual 
presentation of the data. This exploration is necessary as a basis for deci- 
sions about the questions for which answers might reasonably be sought and the 
types of analysis which might most appropriately be used. Through such explo- 
ration, more formal_ models of the phenomena can be developed and their goodness 
of fit tested with both the original and subsequent data. Traditional statis- 
tical analysis is designed more for the latter task of model fitting than for 
the former task of data exploration. Indeed, a confirmatory data analysis 
restricts the analyst's attention to the model and limits _the potential conclu- 
sions about the model to "fitting" or "not fitting." Confirmatory procedures 
give almost no guidance on how to explore data. 

Basic Procedures ; For the stem and leaf display the da ta are grouped into 
categories. The beginning digits of each category become part of the stem. 
Each category starts a new row. The last digits of the scores of each 
category are displayed on the row. These last digits are referred, to as the 
leaves . A box plot is a visual representation of the central tendency and the 

spread of a set of scores. The box locates the middle 50% of the data, 

extending from the 25th percentile to the 75th percentile. The location of 
the median within the box gives some indication of the symmetry of the data. 
Dotted lines extend Outward from the _ box to the ^Location of the scores closest 
to, but inside, what are termed the "two inner fences." The fences are 
boundaries beyond which the occurrence of scores could be considered unusual. 
The innerfence contains 99% of the distribution. Scores outside the inner 
fences are highlighted individually as large dots in the display. The box 
plot thus highlights unusual scores; 

Advantages /Benefit s : The stem and leaf display has two advantages over the 
histogram. _0ne is that it retains all the original data so that the content 
of the groupings within the batch remain apparent, making detailed examination 
easier. The other is that it is easier to prepare than a histogram so that no 
graph need be drawn. The box plot gives ready visual inspection of the 
location and shape of the distribution of the data. The box plot provides a 
summary of the data which is resistant to the influence of deviant data and 
yet does not obscure them. The box plot as a simple graphical display helps 
the evaluator to look at the data in more than one way. It can suggest 
further analyses. The box plot can also be very helpful in the evaluator 1 s 
attempts to connnonicate with others, particularly with lay audiences. 

Disadvantages/Costs . Lay audiences cannot interpret the stem and leaf display 
or the box plot unless they are given prior tutoring in the meaning of the 
stem and leaf display or the box plot. 

Resources Required ; None. 

Basic— R e f e r ences ; 

McGaw, B. Exploratory data analysis. In N. L. Smith (Ed.) # New tech niques— 
for evaluation . Beverly Hills: SAGE Publications, 1981. 

Tukey, J. W. Exploratory data ^analysis . Reading , MA: Addison-Wesley , 1977. 
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Method : Still Photography 

Purpose ; To describe evaluative information by means of photographs. 

Why— and When to Use ; The following is a. listing of conditions for which 
evaluators might choose to use photography: 

• a record of events in detail. 

• visual information that is of primary interests 

• tracking the activities of a single participant in the program. 

• thicfc_<3§3cription of _ the_prpcess_in a program rather than outcomes. 

• a record of information about social interaction of particular 
individuals, groups, or communities. 

• documenting regular routines of regular and special classes or 
programs. 

<r activities rather than goals of a program. 

• understanding the subjective nature of the participants' experience. 
sedures : The procedures for the data collection are as follows: 



Taking pictures. 

Classifying them, focusing the topic, selecting the most relevant. 
Gathering yerbatm da 

Photointerviewing or showing photographs to the subjects for their 
interpretation. 

^ing_^ck_freq^ently_to_conf disconfirra the data, staying in 

the study site as long as possible. 

Selecting and interpreting photographs , and arranging a final report. 



Ad vantage s /B e n efits: Photography constitutes a form of visual communication 
that provides relevant nonverbal information arid basic imagery not available 
by other means. 

Disadvantages/Costs : A photograph is open to a wide range of subjective 
interpretation • 

Resources Required : Photographic equipment. 
Basic References t 

Templin, P. A. Photography as an evaluation technique. No. 32 Of the Paper 
and Report Series of the Research on Evaluation Program, Portland, OR: 
Northwest Regional Educational Laboratory, 1973. 

Templin, P.A. Still photography in evaluation. In N.L. Smith (Ed.)., 
Communication strategies in evaluation . Beverly Hills, CA: SAGE 
Publications, 1982. 
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Method ; Oral Briefings 

Purpose ; An oral briefing is a personal interaction between evaluator and 

audierice in i which the results of evaluation are presented orally. The purpose 
is to increase the impact of the evaluation on the audience. 

Why and Wh e n to U se ; Oral briefings are used when the evaluator wants to have 
more impact on the audience than just a written report would have. 

Basic Procedures ; _.. _ _ . _. 

(1) Planning the briefing . A successful briefing must be carefully 
planned far in advance of the actual briefing date. 

(2) Setting the stage . The first. way is to determine the participants, _ 
both among the presenters and the audience. A second way to set the stage for 
the briefing is to provide advance briefing materials to each audience member. 

(3) Presenting the briefing , fit the beginning the presenter captures the 
audience ' s_attention. _ Then_the presentermust provide three basic types of 
information: (a) a description of the program, (b) a comparison of this _ 
4D#°™§tion with .appropriate o^er information to proyi^ and (c) 
any interpretations of the findings and their implications. 

(4) Following up . Often a briefing can crystallize a number of followap 
actions. _ First, the_ assignments _need_ to be defined clearly and : completely . 
Second, responsibility for each action needs to be assigned to a specific 
individual. Third, staff must be assigned the responsibility Of tracking the 
assigned task and its completion. Fourth, it is necessary for the formal 
assignment to include a specific role for the evaluator. 

Advantages/Benefits : Briefings, being .visible events involving several 
persons, possess an impact that is difficult to ignore, briefings do hot have 
the impersonal exchange of a written report. 

D is advantage 3 /Co 3 t3 ; Briefings are transitory and hot lasting, and thus are 
open to misinterpretation or misrepresentation by audience members. The 
presentation is almost entirely dependent on the personal abilities of the 
single. presenter . The nature of a briefing requires omitting from the 
presentation much useful information. 

Resources Required ; None* 

Basic Reference ; 

Hendricks, M. Oral policy briefings. In N.L. Smith (Ed.), €oroun4oa%ion 
strategies in evaluation . Beverly Hills, CA: SAGE Publications, 1982. 
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Method: Briefing Panel Presentations 

Purpose: A briefing panel is a group of people, perhaps a project's 

advisory committee, called together to hear a compressed presentation of 
projectactiyitiesand findings, its members, raise Questions r discuss issues, 
and perhaps offer additional interpretations and recommendations* The burden 
°!_th«5 eyaiuatqrs is to devise more pungent presentations, anticipate data 
needs, and incorporate reactions. 

WhyandzWhen to Use ; Evaluators can use a briefing panel to learn how better 
to inform others through their evaluation reports. 

Bas i c P rocedur e s : An example of a briefing panel presentation is as follows: 
Pans lists who have some expertise in the area being ovaluated were chosen. 
The panelists were charged: (1) to consider specifically the strengths and 
shortcomings of the program strategy and to consider more generally the 
various strategies and obstacles to such improvement; and __(2) to discuss the 
philosophy, methods, and results to date of the program .being evaluated. 

Proceedings began at 9 a.m. with an hour set aside for presentations by 
evaluators and .witnesses. At 10 a.m. the panelists were asked to break an 
imposed silence and to indicate questions or topics on which they would like 
nip^i.i- 11 ? P^a^iOTf which _they got. At ll_a.m. the panelists were asked to take 
charge and to proceed in any way they chose. They were asked to spend at 
least an hour on their task. The issues raised and the alternative ideas 
proposed were included in the interim evaluation report. 

Ad ^ahtagbs/^ efler^lt& : External panel experts can apply a fresh perspective in 
interactions with program-related personnel, can provide raultidisciplinary 
expertise in project reviews, and can permit _use_of nationally known experts 
who would be too expensive to hire on an extended basis. 

D isadvantages/Costs t The reaction of most participants was that the 

proceeding would have been improved with more structure. Panel members and 
some observers indicatedthey felt confused about what the panel was supposed 
to do and what other persons were to do. 

Resources Required : None. 

Stake, R.E., & Balk, D.E. Briefing panel preJera c; .io*#. In r?.L. Smith (Ed.), 
Commmi cation— strateg ies in ev aluation . Beve^i/ Si:ur. ; w ' . SAGE 
Publications, 1982. 
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Method : Adversary Hearings 

Purpose ; To adjudicate between opposing issues of a program which is 

being evaluated, 

Wh y and Wh e n t o— Sse ; The adversarial model operates with the assumption that 
truth emerges from a hard, but fair fight, in which opposing sides; after 
agreeing upon the issues in contention, present evidence in support of each 
side. The fight is refereed by a neutral figure, and all of the relevant 
evidence is weighed by a neutral person (or persons) bo arrive at a fair 
result. 

B asic Procedu res ; 

The Legislative Phase ; In the legal system, a dispute may arise only when 
there is a rule of law to provide the standards against which an act or series 
of events may be evaluated. The analogy of law in program evaluation is a 
statement of the goals arid objectives of a program. Wo 1 ^ (1975) used an 
elaborate dragnet to .garner complaints in a phase he called "issue 
generation 9 m followed by an issue selection phase. 

_ Retrieval Pr ocess ; Vvolf (1975) turned over the task of issue delimitation 
to two teams of evaluators, one of which presented the case for the existing 

program, and the other the case for alternatives. 

The Trial of Hearing; A trial is held in which alternative sides of an 

issue are presented; 

The Jury or the Pan e. ^; Wolf (1975) selected highly qualified 

professionals representing a variety of viewpoints on the program in 
question. Alternatively, a jury can be selected. 

Advan tagea/Bene fits ; Adversary approaches provide a disciplined method for 
introducing and testing qualitative testimony. Cross-examination provides a 
form of control to reveal the limits of any bit of testimony to support a 
proposition. 

Disadvantages/Costs z the adversarial process, as it has been used so far, is 
time-consuming, expensive, and cumbersome. 

Resources Required ; Time f money and participants. 

Basic Ref e r ences ; 

Lev ine, M. Scientific methbo arid the ;»dver3ary model. Some preliminary 
thoughts. American P s ychologist ^ i*J76 , 219 > 661-677. 

Lev ine , M. Advei :«ry heai lit:; ss . In M.: v Smi th (Ed.) , Communication strategies 
in evaluation, Beverly Sills, CA; SAGE Publications, -\?32. 

Wolf^ R. Trial by : ! vry. /- new evaluation method. I. The Erc?ces3. Phi 
Delta Kappan , 1-V's, 57, 135-187. 
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Method ; Committee Hearings 



Purpose : _ Committee hearings have been used, for two purposes. One .purpose 
is to evaluate an evaluation (i.e. raeta-e valuation) . A hearing is held to 
examine tha evaluative procedures, to test the conclusions reached, and to 
examine the recommendations. Another purpose is valuation, i.e., assignment 
of merit or worth through the evaluative process. 

Why and when to Use : Use when there are contending issues that need to be 
resolved surrounding an evaluation. 

Basic Procedures ; 

Committee Selection : It is quite possible that an existing committee can 
serve in a committee hear ing. School boards, textbook commit tees, curr iculum 
committees^ advisory panels, faculty committees, and executive committees may 
be used. Membership on committees could also be determined by identifying 
what kind of expertise would provide an adequate examination of data treatment 
and interpretations. 

^lea : Spies that can be considered are the committee chairperson, 

committee counsel* and witnesses. 

Stages of Operation j The committee membership should hay« a role in- 
preparing guidelines and in selecting witnesses or identifying the typo of 
witnesses they would like to make _ presentations at the hearing. Witnesses 
should be informed about the operation of the hearings. Prior to the 
hearings, the committee should review the procedures to be used. The actual 
format of the hearing will vary with 'the purpose of the hearing. The final 
phase of the committee work is that of summary. 

ft d van ta ^es/Bene^i^s ; The committee hearing ensures that an evaluation will be 
used, not set aside to gather dust. 

Disadvantages/Costs : The committee hearing can be time-consuming, expensive 
and cumbersome. 

Resources Required ; Time, money and participants. 
Basic Reference : 

Stenzei, N. _ Committee hearings. In_ N.L._Smith_ (Ed.) , Commun ica t ion 

strategies in evaluation. Beverly Hills, CA: SAGE Publications, 1982. 
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Method ; Television presentations of Hearings 

Purpose ; To provide high-*quality information to those who are making 

decisions and implementing programs. Television. presentations have a .major 
purpose , that of providing information and clarifying issues— not of judging 
or of rating individual programs* 

Why and When to Use > ; Hearings are designed to serve ah educational function 
by providing a public forum for discussion of a controversial topic from 
different and often competing perspectives. Many decision makers cannot 
a t tend hear i ng s , so y idea tapes of the p r pceed ing s are circui a ted to 
decisionmakers. Television seems best suited to evaluation studies that have 
a naturalistic mode of inquiry. If the purpose of the evaluation is to tall a 
story^ to describe a ^ dynamic^ compiex_prpgram_in_Pperatipn^_tp trace patterns 
of decisionmakers, or to ascertain people's attitudes and opinions, television 
seems an ideal medium for reporting evaluation results. 

Basic Procedures ; Television is particularly suited to evaluations that end 
with ah event such aa i a courtroom hearing* The hearing is not intended to 
result in a victory for one side or the other. There is no jury present to 
enter a final judgment , and all decisions are left to the viewing audience* 
As with all technology, it is. best to involve professionals who know how to 
produce and direct educational television programs. 

Advantages/Benefits ; Television can; 

• record an event with very little distortion. 

• be used to reach large and scattered audiences. 

« transport. the viewer to a scene (classroom) or gain access to an 
individual (politician, celebrity) not normally accessible to the 
viewer. 

• make complex ' is and abstractions more concrete through the use of 
examples or c strations. 



o personal face-to-face interaction between Viewer and communicator are 
eliminated. 

# Reality can be misrepresented through selective reporting and editing. 

# Costa of production are usually much higher than those for other 
®edia> particularly print media.. 

# Complex phenomena that requires long verbal explanations, statistical 
data, or specialized knowledge, may be difficult to portray visually. 

Resources Required: Access to television equipment and personnel. Costs may 
range into the hundreds of thousands of dollars* 

Shoemaker, J.S. Television presentations. In N.I*. Smith (Ed.) , Communication 
strategies in evaluation. Beverly Hills, CA; SAGE publications, 1982. 
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